Computer system providing hierarchical display remoting optimized with user and system hints and related methods

ABSTRACT

A computing device may include a memory and a processor cooperating with the memory to host virtual computing sessions to be remotely displayed at a client device via a frame buffer, with the client device being configured to render the virtual computing sessions via a graphical user interface (GUI). The computing device may further detect regions of interactive graphics within a virtual computing session based upon a virtual session input and without analyzing the frame buffer, assign a higher priority to the regions of interactive graphics than other content within the frame buffer, and send the contents of the frame buffer to the client device for rendering in the GUI based upon the assigned priority.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. provisional app. No.62/667,072 filed May 4, 2018, which is hereby incorporated herein in itsentirety by reference.

BACKGROUND

Traditionally, personal computers include combinations of operatingsystems, applications, and user settings, which are each managedindividually by owners or administrators on an ongoing basis. However,many organizations are now using application and/or desktopvirtualization to provide a more flexible option to address the varyingneeds of their users. In desktop virtualization, a user's computingenvironment (e.g., operating system, applications, and/or user settings)may be separated from the user's physical computing device (e.g.,smartphone, laptop, desktop computer). Using client-server technology, a“virtualized desktop” may be stored in and administered by a remoteserver, rather than in the local storage of the client computing device.

There are several different types of desktop virtualization systems. Asan example, Virtual Desktop Infrastructure (VDI) refers to the processof running a user desktop inside a virtual machine that resides on aserver. VDI and other server-based desktop virtualization systems mayprovide personalized desktops for each user, while allowing forcentralized management and security. Servers in such systems may includestorage for virtual desktop images and system configuration information,as well as software components to provide the virtual desktops and allowusers to interconnect to them. For example, a VDI server(s) may includeone or more hypervisors (virtual machine managers) to create andmaintain multiple virtual machines, software to manage thehypervisor(s), a connection broker, and software to provision and managethe virtual desktops. In some embodiments, a VDI server(s) may provideaccess to shared server-based hosted applications, as well asWeb/Software-as-a-Service (SaaS) applications.

Desktop virtualization systems may be implemented using a singlevirtualization server or a combination of servers interconnected as aserver grid. For example, a cloud computing environment, or cloudsystem, may include a pool of computing resources (e.g., desktopvirtualization servers), storage disks, networking hardware, and otherphysical resources that may be used to provision virtual desktops,and/or provide access to shared applications, along with additionalcomputing devices to provide management and customer portals for thecloud system. In some implementations, Desktop as a Service (DaaS)sessions may be run from a cloud computing environment for differenttenants or enterprises.

SUMMARY

A computing device may include a memory and a processor cooperating withthe memory to host virtual computing sessions to be remotely displayedat a client device via a frame buffer, with the client device beingconfigured to render the virtual computing sessions via a graphical userinterface (GUI). The virtualization server may further detect regions ofinteractive graphics within a virtual computing session based upon avirtual session input and without analyzing the frame buffer, assign ahigher priority to the regions of interactive graphics than othercontent within the frame buffer, and send the contents of the framebuffer to the client device for rendering in the GUI based upon theassigned priority.

In an example implementation, the computing device may be configured tocapture the virtual session input. By way of example, the virtualsession input may comprise one or more of a window pattern event, ascroll pattern event, and a text pattern event. In accordance with oneexample, the processor may be further configured to asynchronously sendthe text pattern input data to the client device for rendering prior torendering of corresponding contents of the frame buffer. Moreparticularly, the text pattern input data may comprise text data andassociated text metadata, and the text metadata may comprise at leastone of font type, font size, foreground color, background color, italic,bold, underline, and line spacing, for example. In still another exampleimplementation, the text pattern input data may correspond to a textpattern within a scroll container.

Furthermore, the processor may be configured to send the contents of theframe buffer corresponding to the regions of interest with a differentlevel of forward error correction (FEC) than other content within theframe buffer. In an example embodiment, the processor may be configuredto send graphics data corresponding to the regions of interactivegraphics from the frame buffer to the client computing device via afirst channel, and send the remaining graphic data from the frame bufferto the client computing device via a second channel. In addition, theprocessor may also be configured to send graphics data corresponding tothe regions of interactive graphics via a different virtual channel thanother content within the frame buffer, for example.

A related method may include hosting virtual computing sessions at acomputing device to be remotely displayed at a client device via a framebuffer, with the client computing device being configured to render thevirtual computing sessions via a GUI. The method may further includedetecting regions of interactive graphics within a virtual computingsession at the computing device based upon a virtual session input andwithout analyzing the frame buffer, assigning a higher priority to theregions of interactive graphics than other content within the framebuffer at the computing device, and sending the contents of the framebuffer from the computing device to the client device for rendering inthe GUI based upon the assigned priority.

A related computing system may include a server configured to hostvirtual computing sessions and remotely display the virtual computingsessions via a frame buffer, and a client device configured to remotelyaccess virtual computing sessions from the server via a GUI. The servermay be further configured to detect regions of interactive graphicswithin a virtual computing session based upon a virtual session inputand without analyzing the frame buffer, assign a higher priority to theregions of interactive graphics than other content within the framebuffer, and send the contents of the frame buffer to the client devicefor rendering in the GUI based upon the assigned priority.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a network environment of computing devicesin which various aspects of the disclosure may be implemented.

FIG. 2 is a block diagram of a computing device useful for practicing anembodiment of the client machines or the remote machines illustrated inFIG. 1.

FIG. 3 is a schematic block diagram of a cloud computing environment inwhich various aspects of the disclosure may be implemented.

FIG. 4 is a diagram illustrating round trip times (RTT) for differentcloud services Points of Presence (POPs) based upon geographic location.

FIG. 5 is a schematic block diagram of an example computing system inwhich various embodiments may be implemented for providing virtualcomputing sessions with enhanced client computing device displayremoting features.

FIG. 6 is a schematic block diagram of a computing system which providesenhanced display remoting through interactive graphics prioritization inaccordance with an example embodiment.

FIG. 7 is a schematic block diagram of the computing system of FIG. 6illustrating further aspects that may be performed with interactivegraphics prioritization in accordance with example embodiments.

FIG. 8 is a flow diagram illustrating method aspects associated with thesystem of FIG. 6.

FIG. 9 is a schematic block diagram of a computing system which providesenhanced display remoting through forward error correction (FEC) inaccordance with an example embodiment.

FIG. 10 is a schematic block diagram of the computing system of FIG. 9illustrating further aspects that may be performed with FEC inaccordance with example embodiments.

FIG. 11 is a flow diagram illustrating method aspects associated withthe system of FIG. 9.

FIG. 12 is a schematic block diagram of a computing system whichprovides enhanced display remoting through user input prioritization andtransmission in accordance with an example embodiment.

FIG. 13 is a schematic block diagram of the computing system of FIG. 12illustrating further aspects that may be performed with user inputprioritization and transmission in accordance with example embodiments.

FIG. 14 is a flow diagram illustrating method aspects associated withthe system of FIG. 12.

DETAILED DESCRIPTION

Generally speaking, the embodiments disclosed herein relate to computingsystems providing remote access to virtual computing sessions. Moreparticularly, as more workloads are migrated into the Cloud anddatacenters are consolidated across continents, the network conditionsbetween the workloads and the client endpoints become more challengingand are characterized with much higher latency and packet loss, makingit difficult to remotely display virtual computing sessions at clientdevices. Moreover, high-resolution, e.g. 4K, and multi-monitor displaysmake graphics loads even higher. Conversely, while using mobile devices,while resolutions are lower, there is expectation of native-likeapplication responsiveness with Citrix Receiver/Workspace App.

Current user experience with respect to remotely displayed virtualcomputing sessions may accordingly become “laggy” in certain instances.For examples, such instances may include: normal office workloads suchas typing, mouse or touch window dragging, scrolling, app switching,etc.; 3D Pro use cases of interacting with complex 3D graphics models;and Virtual Reality (VR) use cases with XenApp hosted apps.

The embodiments set forth herein advantageously provide for enhanceddelivery and rendering of graphics from virtual sessions hosted by avirtualization server, which in some implementations may be hosted in acloud computing environment. This results in an improved quality and/orspeed at which user input may be delivered to virtual computing sessionsand/or graphics updates may be rendered in a graphical user interface(GUI) by client computing devices, for example, with reduced ornegligible impact to bandwidth and/or system load.

The present description is made with reference to the accompanyingdrawings, in which example embodiments are shown. However, manydifferent embodiments may be used, and thus the description should notbe construed as limited to the particular embodiments set forth herein.Like numbers refer to like elements throughout.

As will be appreciated by one of skill in the art upon reading thefollowing disclosure, various aspects described herein may be embodiedas a device, a method or a computer program product (e.g., anon-transitory computer-readable medium having computer executableinstruction for performing the noted operations or steps). Accordingly,those aspects may take the form of an entirely hardware embodiment, anentirely software embodiment, or an embodiment combining software andhardware aspects.

Furthermore, such aspects may take the form of a computer programproduct stored by one or more computer-readable storage media havingcomputer-readable program code, or instructions, embodied in or on thestorage media. Any suitable computer readable storage media may beutilized, including hard disks, CD-ROMs, optical storage devices,magnetic storage devices, and/or any combination thereof.

Referring initially to FIG. 1, a non-limiting network environment 101 inwhich various aspects of the disclosure may be implemented includes oneor more client machines 102A-102N, one or more remote machines106A-106N, one or more networks 104, 104′, and one or more appliances108 installed within the computing environment 101. The client machines102A-102N communicate with the remote machines 106A-106N via thenetworks 104, 104′.

In some embodiments, the client machines 102A-102N communicate with theremote machines 106A-106N via an intermediary appliance 108. Theillustrated appliance 108 is positioned between the networks 104, 104′and may also be referred to as a network interface or gateway. In someembodiments, the appliance 108 may operate as an application deliverycontroller (ADC) to provide clients with access to business applicationsand other data deployed in a datacenter, the cloud, or delivered asSoftware as a Service (SaaS) across a range of client devices, and/orprovide other functionality such as load balancing, etc. In someembodiments, multiple appliances 108 may be used, and the appliance(s)108 may be deployed as part of the network 104 and/or 104′.

The client machines 102A-102N may be generally referred to as clientmachines 102, local machines 102, clients 102, client nodes 102, clientcomputers 102, client devices 102, computing devices 102, endpoints 102,or endpoint nodes 102. The remote machines 106A-106N may be generallyreferred to as servers 106 or a server farm 106. In some embodiments, aclient device 102 may have the capacity to function as both a clientnode seeking access to resources provided by a server 106 and as aserver 106 providing access to hosted resources for other client devices102A-102N. The networks 104, 104′ may be generally referred to as anetwork 104. The networks 104 may be configured in any combination ofwired and wireless networks.

A server 106 may be any server type such as, for example: a file server;an application server; a web server; a proxy server; an appliance; anetwork appliance; a gateway; an application gateway; a gateway server;a virtualization server; a deployment server; a Secure Sockets LayerVirtual Private Network (SSL VPN) server; a firewall; a web server; aserver executing an active directory; a cloud server; or a serverexecuting an application acceleration program that provides firewallfunctionality, application functionality, or load balancingfunctionality.

A server 106 may execute, operate or otherwise provide an applicationthat may be any one of the following: software; a program; executableinstructions; a virtual machine; a hypervisor; a web browser; aweb-based client; a client-server application; a thin-client computingclient; an ActiveX control; a Java applet; software related to voiceover internet protocol (VoIP) communications like a soft IP telephone;an application for streaming video and/or audio; an application forfacilitating real-time-data communications; a HTTP client; a FTP client;an Oscar client; a Telnet client; or any other set of executableinstructions.

In some embodiments, a server 106 may execute a remote presentationservices program or other program that uses a thin-client or aremote-display protocol to capture display output generated by anapplication executing on a server 106 and transmit the applicationdisplay output to a client device 102.

In yet other embodiments, a server 106 may execute a virtual machineproviding, to a user of a client device 102, access to a computingenvironment. The client device 102 may be a virtual machine. The virtualmachine may be managed by, for example, a hypervisor, a virtual machinemanager (VMM), or any other hardware virtualization technique within theserver 106.

In some embodiments, the network 104 may be: a local-area network (LAN);a metropolitan area network (MAN); a wide area network (WAN);software-defined networking in a wide area network (SD-WAN); a primarypublic network 104; and a primary private network 104. Additionalembodiments may include a network 104 of mobile telephone networks thatuse various protocols to communicate among mobile devices. For shortrange communications within a wireless local-area network (WLAN), theprotocols may include 802.11, Bluetooth, and Near Field Communication(NFC).

One particularly advantageous implementation of SD-WAN enterprises isprovided by Citrix SD-WAN, which allows enterprises to render their WANswith more scalability, and in a cost-effective that is ready to connectto the cloud. Citrix SD-WAN contains an integrated database and deeppacket inspection to identify applications, including individual SaaSapplications, and intelligently steer traffic from the branch to theinternet, cloud, or SaaS. Moreover, Citrix SD-WAN also provides theability to route traffic from the branch to the internet via a secureweb gateway, delivering cloud-based security including firewall, URLfiltering, and usage accounting.

FIG. 2 depicts a block diagram of a computing device 100 useful forpracticing an embodiment of client devices 102, appliances 108 and/orservers 106. The computing device 100 includes one or more processors103, volatile memory 122 (e.g., random access memory (RAM)),non-volatile memory 128, user interface (UI) 123, one or morecommunications interfaces 118, and a communications bus 150.

The non-volatile memory 128 may include: one or more hard disk drives(HDDs) or other magnetic or optical storage media; one or more solidstate drives (SSDs), such as a flash drive or other solid-state storagemedia; one or more hybrid magnetic and solid-state drives; and/or one ormore virtual storage volumes, such as a cloud storage, or a combinationof such physical storage volumes and virtual storage volumes or arraysthereof.

The user interface 123 may include a graphical user interface (GUI) 124(e.g., a touchscreen, a display, etc.) and one or more input/output(I/O) devices 126 (e.g., a mouse, a keyboard, a microphone, one or morespeakers, one or more cameras, one or more biometric scanners, one ormore environmental sensors, and one or more accelerometers, etc.).

The non-volatile memory 128 stores an operating system 115, one or moreapplications 116, and data 117 such that, for example, computerinstructions of the operating system 115 and/or the applications 116 areexecuted by processor(s) 103 out of the volatile memory 122. In someembodiments, the volatile memory 122 may include one or more types ofRAM and/or a cache memory that may offer a faster response time than amain memory. Data may be entered using an input device of the GUI 124 orreceived from the I/O device(s) 126. Various elements of the computer100 may communicate via the communications bus 150.

The illustrated computing device 100 is shown merely as an exampleclient device or server, and may be implemented by any computing orprocessing environment with any type of machine or set of machines thatmay have suitable hardware and/or software capable of operating asdescribed herein.

The processor(s) 103 may be implemented by one or more programmableprocessors to execute one or more executable instructions, such as acomputer program, to perform the functions of the system. As usedherein, the term “processor” describes circuitry that performs afunction, an operation, or a sequence of operations. The function,operation, or sequence of operations may be hard coded into thecircuitry or soft coded by way of instructions held in a memory deviceand executed by the circuitry. A processor may perform the function,operation, or sequence of operations using digital values and/or usinganalog signals.

In some embodiments, the processor can be embodied in one or moreapplication specific integrated circuits (ASICs), microprocessors,digital signal processors (DSPs), graphics processing units (CPUs),microcontrollers, field programmable gate arrays (FPGAs), programmablelogic arrays (PLAs), multi-core processors, or general-purpose computerswith associated memory.

The processor 103 may be analog, digital or mixed-signal. In someembodiments, the processor 103 may be one or more physical processors,or one or more virtual (e.g., remotely located or cloud) processors. Aprocessor including multiple processor cores and/or multiple processorsmay provide functionality for parallel, simultaneous execution ofinstructions or for parallel, simultaneous execution of one instructionon more than one piece of data.

The communications interfaces 118 may include one or more interfaces toenable the computing device 100 to access a computer network such as aLAN, a WAN or SD-WAN, a Personal Area Network (PAN), or the Internetthrough a variety of wired and/or wireless connections, includingcellular connections.

In described embodiments, the computing device 100 may execute anapplication on behalf of a user of a client device. For example, thecomputing device 100 may execute one or more virtual machines managed bya hypervisor. Each virtual machine may provide an execution sessionwithin which applications execute on behalf of a user or a clientdevice, such as a hosted desktop session. The computing device 100 mayalso execute a terminal services session to provide a hosted desktopenvironment. The computing device 100 may provide access to a remotecomputing environment including one or more applications, one or moredesktop applications, and one or more desktop sessions in which one ormore applications may execute.

An example virtualization server 106 may be implemented using CitrixHypervisor provided by Citrix Systems, Inc., of Fort Lauderdale, Fla.(“Citrix Systems”). Virtual app and desktop sessions may further beprovided by Citrix Virtual Apps and Desktops, also from Citrix Systems.Citrix Virtual Apps and Desktops is an application virtualizationsolution that enhances productivity with universal access to virtualsessions including virtual app, desktop, and data sessions from anydevice, plus the option to implement a scalable VDI solution.

Referring to FIG. 3, a cloud computing environment 160 is depicted,which may also be referred to as a cloud environment, cloud computing orcloud network. The cloud computing environment 160 can provide thedelivery of shared computing services and/or resources to multiple usersor tenants. For example, the shared resources and services can include,but are not limited to, networks, network bandwidth, servers,processing, memory, storage, applications, virtual machines, databases,software, hardware, analytics, and intelligence.

In the cloud computing environment 160, one or more clients 162A-162C(such as those described above) are in communication with a cloudnetwork 164. The cloud network 164 may include back-end platforms, e.g.,servers, storage, server farms or data centers. The users or clients162A-162C can correspond to a single organization/tenant or multipleorganizations/tenants. More particularly, in one example implementationthe cloud computing environment 160 may provide a private cloud servinga single organization (e.g., enterprise cloud). In another example, thecloud computing environment 160 may provide a community or public cloudserving multiple organizations/tenants.

In some embodiments, a gateway appliance(s) or service may be utilizedto provide access to cloud computing resources and virtual sessions. Byway of example, Citrix Gateway, provided by Citrix Systems, Inc., may bedeployed on-premises or on public clouds to provide users with secureaccess and single sign-on to virtual, SaaS and web applications.Furthermore, to protect users from web threats, a gateway such as CitrixSecure Web Gateway may be used. Citrix Secure Web Gateway uses acloud-based service and a local cache to check for URL reputation andcategory.

In still further embodiments, the cloud computing environment 160 mayprovide a hybrid cloud that is a combination of a public cloud and aprivate cloud. Public clouds may include public servers that aremaintained by third parties to the clients 162A-162C or theenterprise/tenant. The servers may be located off-site in remotegeographical locations or otherwise.

The cloud computing environment 160 can provide resource pooling toserve multiple users via clients 162A-162C through a multi-tenantenvironment or multi-tenant model with different physical and virtualresources dynamically assigned and reassigned responsive to differentdemands within the respective environment. The multi-tenant environmentcan include a system or architecture that can provide a single instanceof software, an application or a software application to serve multipleusers. In some embodiments, the cloud computing environment 160 canprovide on-demand self-service to unilaterally provision computingcapabilities (e.g., server time, network storage) across a network formultiple clients 162A-162C. By way of example, provisioning services maybe provided through a system such as Citrix Provisioning Services(Citrix PVS). Citrix PVS is a software-streaming technology thatdelivers patches, updates, and other configuration information tomultiple virtual desktop endpoints through a shared desktop image. Thecloud computing environment 160 can provide an elasticity to dynamicallyscale out or scale in responsive to different demands from one or moreclients 162. In some embodiments, the cloud computing environment 160can include or provide monitoring services to monitor, control and/orgenerate reports corresponding to the provided shared services andresources.

In some embodiments, the cloud computing environment 160 may providecloud-based delivery of different types of cloud computing services,such as Software as a service (SaaS) 170, Platform as a Service (PaaS)172, Infrastructure as a Service (IaaS) 174, and Desktop as a Service(DaaS) 176, for example. IaaS may refer to a user renting the use ofinfrastructure resources that are needed during a specified time period.IaaS providers may offer storage, networking, servers or virtualizationresources from large pools, allowing the users to quickly scale up byaccessing more resources as needed. Examples of IaaS include AMAZON WEBSERVICES provided by Amazon.com, Inc., of Seattle, Wash., RACKSPACECLOUD provided by Rackspace US, Inc., of San Antonio, Tex., GoogleCompute Engine provided by Google Inc. of Mountain View, Calif., orRIGHTSCALE provided by RightScale, Inc., of Santa Barbara, Calif.

PaaS providers may offer functionality provided by IaaS, including,e.g., storage, networking, servers or virtualization, as well asadditional resources such as, e.g., the operating system, middleware, orruntime resources. Examples of PaaS include WINDOWS AZURE provided byMicrosoft Corporation of Redmond, Wash., Google App Engine provided byGoogle Inc., and HEROKU provided by Heroku, Inc. of San Francisco,Calif.

SaaS providers may offer the resources that PaaS provides, includingstorage, networking, servers, virtualization, operating system,middleware, or runtime resources. In some embodiments, SaaS providersmay offer additional resources including, e.g., data and applicationresources. Examples of SaaS include GOOGLE APPS provided by Google Inc.,SALESFORCE provided by Salesforce.com Inc. of San Francisco, Calif., orOFFICE 365 provided by Microsoft Corporation. Examples of SaaS may alsoinclude data storage providers, e.g. Citrix ShareFile from CitrixSystems, DROPBOX provided by Dropbox, Inc. of San Francisco, Calif.,Microsoft SKYDRIVE provided by Microsoft Corporation, Google Driveprovided by Google Inc., or Apple ICLOUD provided by Apple Inc. ofCupertino, Calif.

Similar to SaaS, DaaS (which is also known as hosted desktop services)is a form of virtual desktop infrastructure (VDI) in which virtualdesktop sessions are typically delivered as a cloud service along withthe apps used on the virtual desktop. Citrix Cloud from Citrix Systemsis one example of a DaaS delivery platform. DaaS delivery platforms maybe hosted on a public cloud computing infrastructure such as AZURE CLOUDfrom Microsoft Corporation of Redmond, Wash. (herein “Azure”), or AMAZONWEB SERVICES provided by Amazon.com, Inc., of Seattle, Wash. (herein“AWS”), for example. In the case of Citrix Cloud, Citrix Workspace appmay be used as a single-entry point for bringing apps, files anddesktops together (whether on-premises or in the cloud) to deliver aunified experience.

Turning now to FIGS. 4-5, systems and methods are provided forhierarchical display within a remote computing context, includingoptimized region detection and delivery in challenging networkconditions using a combination of user and system hints. By way ofbackground, as more workloads are migrated into the Cloud anddatacenters are consolidated across continents, the network conditionsbetween the workloads and the client endpoints become more challengingand are characterized with much higher latency and packet loss. Forexample, Citrix Workspace is a service that provides consolidated remoteaccess to cloud-based workloads of desktops and apps, e.g. CitrixVirtual Apps and Desktops (CVAD) among other services, or a hybridcombination of Cloud and on-premises workloads with consolidated Storeexperience. In addition, Citrix Gateway as a Service (GWaaS), allowsusers to remotely access both Cloud and on-premises workloads withoutsetting up customer on-premises Citrix NetScaler Gateway. However, thenetwork hops to and from the Cloud further increase network latencies.

The bar chart 40 of FIG. 4 shows the Independent Computing Architecture(ICA) file Round Trip Time (RTT) in the different Citrix Gateway Pointsof Presence (POPs) based on geographic location. More particularly,Citrix ICA is a proprietary protocol for an application server system.It is not bound to any single platform, and provides a specification forpassing data between server and clients. Citrix ICA includes a serversoftware component, a network protocol component, and a client softwarecomponent. All measurements are based on Citrix High-definition UserExperience (HDX) over Transmission Control Protocol (TCP) as transportprotocol. Citrix HDX technology is a set of protocols and capabilitiesthat work together to deliver a high-definition user experience ofvirtual desktops and applications to client computing devices over anynetwork from a data center. ICA RTT measures the interactivity of theHDX session, e.g., the click-to-photon time. For example, this may bethe time difference between a user input event (keyboard, mouse, touch,electronic pen) at a client device hosting Citrix Receiver or CitrixWorkspace App (CWA) and the corresponding graphics update generated bythe hosted application and delivered at the client device hosting CWA.ICA RTT is not necessarily equal to the network layer 7 latency, becauseICA RTT can be influenced by the application response time, server load,etc.

As seen in the bar chart 40, user experience (UX) with the Asia-Pacificworkloads (as-aria-se) is particularly challenging. But even in the US,over 50% of the connections incur ICA RTT of 100 ms or higher, e.g.,az-us-w or az-us-sc. There are additional cases of users in remotebranch office locations using Very Small Aperture Terminal (VSAT)connections with 1000-2000 ms latency. In the bar chart 40, “aws” refersto Amazon Web Services installations, while “az” refers Azureinstallations.

In addition, high-resolution (e.g. 4K) and multi-monitor displays makegraphics loads even higher. Conversely, when using mobile devices, whileresolutions are lower, there may be an expectation of native-likeapplication responsiveness with products such as CitrixReceiver/Workspace App. User experience may be “laggy” in certainsituations, such as: normal office workloads such as typing, mouse ortouch window dragging, scrolling, app switching, etc.; 3D Pro use casesof interacting with complex 3D graphics models; and Virtual Reality (VR)use cases with virtual hosted apps, e.g. with apps and desktops providedby CVAD.

To improve both throughput and interactivity, example embodiments of thesystems and methods set forth herein may utilize a shift to UserDatagram Protocol (UDP) as a transport protocol to overcome some of thelimitations of TCP. TCP uses the Additive Increments MultiplicativeDecrements (AIMD) congestion control algorithm and cannot fully utilizebandwidth in conditions of latency and loss. In modern networksbandwidth tends to be more plentiful and packet loss is mostly due toend-point interference (stochastic loss) and not congestion. Therefore,some packet loss can be tolerated without drastically reducingthroughput. For example, entire Google product families are migrating toQUIC. Citrix HDX has recently introduced the Enlightened Data Transport(EDT) protocol, a reliable-transmission UDP-based protocol, which hasbetter congestion and flow control than TCP in challenging networkconditions, while still being fair in terms of bandwidth allocation.

For a typical transoceanic WAN of 250 ms RTT and 1% packet loss eachway, HDX over EDT as transport performs much better than over TCP asfollows: the virtual session interactivity is up to 2.5 times better(the extent depends on the workload, e.g., scrolling text in Notepad vs.web page with images); the speed of file transfer with client drivemapping (remote file copy)—up to 10× improvements; the speed of printing(printing from a hosted application to a client or network-mappedprinter)—up to 2× improvements; and the speed of generic USBredirection—up to 35% improvements. The more stream-oriented and lessinteractive the remoting technology, the higher the benefits of reliableEDT. Therefore, the performance improvements with EDT vary with thedifferent technologies being remoted and the respective remoting virtualchannel protocols.

In each of the bars in the bar chart 40, the bottom section of the barrepresents the percentage of ICA files that have an RTT time <50 ms; thenext section up is the percentage of ICA files that have an RTT time of50-100 ms; the next section up is the percentage of ICA files that havean RTT time of 100-200 ms; the next section up is the percentage of ICAfiles that have an RTT time of 200-450 ms; and the top section of eachbar represents the percentage of ICA files that have an RTT time of >450ms, as indicated by the associated key.

While EDT benefits all ICA Virtual Channels (VCs) to varying degrees,there is still room for improvement due to the fact that EDT iscurrently a reliable protocol (only)—meaning any lost packets results inpacket retransmissions—which invariably limits interactivity of UX. Forexample, loss of user input packets could cause delay in transmission ofsubsequent user input packets until the lost packets are successfullyretransmitted. As another example, loss of graphics update packets couldcause delay in transmission of subsequent graphics update packets untilthe lost packets are successfully retransmitted. Improvements in HDXThinwire such as progressive display for transient regions (fuzzy-first,sharpen up when changes stop) help improve UX, but they are stilllimited by the reliable nature of the underlying transport protocol.Thinwire is the Citrix default display remoting technology used in CVAD.Display remoting technology allows graphics generated on one machine tobe transmitted, typically across a network, to another machine fordisplay.

In addition, current graphics region analysis and classification isbased on framebuffer analysis, which is CPU and memory intensive. WhileAPIs such as the Microsoft Desktop Deduplication API provide facilitiesfor “dirty region” detection (e.g. graphics regions that have recentlychanged and require and update to be transmitted), they are notavailable on all OSs and graphics driver models in use by CVAD, forexample. In addition, even when available, these APIs do not provideadditional granular information about the semantics of the graphicsdata, for example, a distinction between static images, fast changinggraphics (video), text. Therefore, additional framebuffer analysis isalways required for efficient operation.

Referring to FIG. 5, the system 50 illustrates a display remotingarchitecture by which client devices 51 may remotely access virtualcomputing sessions (e.g., virtual desktops and apps) from a host, suchas a VDI or cloud-based virtualization infrastructure. In theillustrated example, the client devices 51 are connected to the virtualcomputing sessions by a virtual delivery appliance 52, such as providedby Citrix VDA, for example. The client device 51 and virtual deliveryappliance 52 communicate via a plurality of different virtual channels,including a Thinwire (Graphics) channel 53, Client Drive Mapping (CDM)channel 54, Universal Serial Bus (USB) channel 55, multimedia channel56, etc. Virtual channel data passes through a series of drivers withinan Independent Computing Architecture (ICA) protocol stack 49 whichperforms different functions, by loading a set of required and optionalprotocol drivers to establish a successful and fully functionalconnection to a remote session, including a winstation driver (WD) 57which performs virtual channel multiplexing, compression,prioritization, and bandwidth limiting, for example. Further, one ormore instances of a protocol driver (PD) 58 may perform, for example, IPframing and secure ICA (custom encryption) operations, while a transportdriver (TD) 59 performs TCP stack operations such as Citrix CommonGateway Protocol (CGP) or SOCKSS, WebSockets, TLS, and TCP, and/or UDPstack operations including CGP, Enlightened Data Transport (EDT) (withoptions for both reliable and unreliable (lossy) transport), DatagramTransport Layer Security (DTLS), and UDP.

In the illustrated example, between the winstation driver 57 andprotocol driver 58, the ICA packets are payload bytes in compressedformat, and between the protocol driver and transport driver 59 the ICApackets have framing and/or internal encryption. Between the transportdriver 59 and a network link (which may include switches, routers,firewalls, gateways, traffic shapers, etc.), the ICA packets are in awire format with CGP headers, transport level encryption (if enabled),ethernet headers, etc. The same operations illustrated on the virtualdelivery appliance 52 side of the system 50 are also performed by thecorresponding components on the client side. It should be noted that thesystem 50 is but one example configuration in which the example displayremoting techniques set forth herein may be implemented, and that othersuitable architectures may be used in different implementations.

Generally speaking, various advantageous display remoting optimizationtechniques may be performed within the example architecture of thesystem 50 (or other suitable architectures). These include optimizationtechniques for region detection, classification and updateprioritization, and hierarchical display with optimized graphicsremoting in challenging network conditions (with latency and/or packetloss). More particularly, the optimized graphics remoting may help to:achieve better interactivity at the expense of fidelity with lossytransmission or combination of reliable and lossy transmission in someareas; further improve interactivity in important areas with ForwardError Correction (FEC) of reliable transmissions (with some trade offwith bandwidth and CPU utilization); regain fidelity in important areaswith FEC of lossy transmissions (and thus achieving better quality alongwith interactivity at the expense of some bandwidth and CPU); andselecting optimal FEC algorithm based on expected bitrate, percentageloss and/or other factors. In addition, in some implementations user andsystem hints may be used to facilitate all of the above, and user inputmay be sent with aggressive FEC to avoid retransmission delays and withnegligible impact on bandwidth. These various aspects will be describedfurther below with respect to FIGS. 6-14.

The example embodiments set forth herein advantageously provide anopportunity to improve user experience through one or more aspects. Afirst aspect is by way of more efficient and granular graphics regiondetection, classification and update prioritization based on combinationof virtual session input from user and system hints, and without havingto perform frame buffer analysis. Another aspect pertains to sendingless important or transient updates in an unreliable (lossy) way over anew EDT-lossy transport protocol, e.g., sending updates in a“fire-and-forget” manner. Still another aspect pertains to sendingimportant updates reliably, or in a lossy way, but in both cases usingFEC. Thus, for reliable traffic, retransmission need only be used whenFEC fails. For lossy traffic, retransmission when FEC fails is optionaldepending on data semantics. For example, if lost, a graphics frame maynot have to be retransmitted because subsequent frames may carry moreup-to-date graphics, thus eliminating the need to retransmit the nowobsolete graphics frame. As another similar example, a lost individualmouse-drag event might not have to be retransmitted because subsequentmouse-drag events may carry a current mouse position, etc. Yet anotheraspect utilizes FEC for both graphics updates and user input to avoidretransmission delays in either direction.

One challenge relates to selection of the FEC algorithm to minimizeoverall impact on bandwidth, or conversely the effective contentbitrate, and CPU and to avoid latency in the FEC computationsthemselves. For example, FEC algorithms work best when applied toequal-sized packets and to constant bit-rate streams such as audio orvideo. However, when applied to interactive traffic such as HDX, thesize of the data packets and the bit-rate are not guaranteed and canvary substantially depending on user activity, application type, sessionresolution, etc. In this regard, various factors may be considered inorder to apply FEC techniques in a useful manner as further describedbelow. These factors include congestion avoidance, i.e., FEC increasesthe bandwidth consumption and/or lowers the effective bitrate of datatransmission, while decreasing effective loss. Yet, sending at a bitrateabove the estimated available bandwidth may cause congestion. Anotherfactor is cost. For example, while bandwidth is more plentiful in modernnetworks, data shows that in some Asia regions it is multiple-times moreexpensive than in the US.

Still another factor to consider is interactivity. If a next packet ismissing from the send queue (for example, if the user has not generatedadditional input events (e.g. keyboard, mouse, touch, pen input) or thehosted application has not generated further graphics updates (e.g. nofurther updates currently required)), to complete a batch of FEC-encodedpackets, a simple application of FEC might have to send dummy packets,which wastes bandwidth, or wait for a next packet, which introducesdelay and defeats the purpose of using FEC. Therefore, predicting abitrate and applying desired FEC encoding for interactive traffic may besignificant for desired performance.

Referring now to FIGS. 6-7, an example virtualization system 60 is shownin which a virtualization server 61 illustratively includes a processor62 configured to host virtual computing sessions 63 to be displayed atone or more client computing devices 64 via a frame buffer 65. As notedabove, the client computing device 64 may take various forms, such as adesktop/laptop computer, smartphone, tablet computer, etc., and isconfigured to render the virtual computing sessions via a graphical userinterface (GUI) 66. The virtualization server 61 may further detectregions of interactive graphics within a virtual computing session 63based upon a virtual session input, which may include user and/or systemhints, without analyzing the frame buffer 65, and assign a higherpriority to the regions of interactive graphics within the frame buffer.As such, the contents of the frame buffer 65 may be sent to the clientcomputing device 64 over a virtual channel(s) 67 for rendering in theGUI based upon the assigned priority. That is, the regions orinteractive graphics may be rendered/updated first as compared to otherareas of the GUI (e.g., static backgrounds, etc.) to provide fasterdisplay updates with respect to the areas of interest to the user,thereby enhancing user experience. Moreover, since the frame buffer 65does not have to be analyzed or scanned line by line to differentiateareas of higher importance (e.g., interactive graphics regions) withinthe frame, this too may contribute to faster processing speeds and/orreduced computational overhead.

In the example implementation of FIG. 7, a UI automation client agent isprovided to capture the virtual session input, including hints takenfrom user input. In other embodiments, the hints taken from user inputmay not require the use of an automation client agent. For example, userinput hints may be directly derived from a virtualization server 61subsystem (not shown) responsible for injecting the user input providedby the client computing device 64 into the respective virtual computingsession 63. Other types of virtual session input may be system hintsgleaned from hooks into the virtual computing sessions 63. Generallyspeaking, the virtual session input may comprise one or more of a windowpattern event, a scroll pattern event, and a text pattern event, forexample.

For example, and by way of a background, UI elements may expose (orsupport) one or more patterns: window pattern (e.g. supported byapplication windows), text pattern (e.g. supported by edit controls,document or rich text viewing and processing controls), scroll pattern(e.g. supported by scrollable views), grid pattern (e.g. supported bytable or spread-sheet processing views). In some instances, UI elementsmay be embedded or otherwise form a hierarchy of UI elements, e.g. adocument editing view may contain a scroll container comprising a textpattern within a scroll pattern. The different UI element patterns mayissue respectively named events. For example, a UI element supporting awindow pattern may issue Window pattern events, e.g. move, maximize,minimize, close, etc. As another example, a UI element supporting a Textpattern may issue Text pattern events, e.g. text changed, text selectionchanged, etc. As yet another example, a UI element supporting a scrollpattern may issue scroll events, e.g. vertical-scroll-up orvertical-scroll-down event, etc.

As will be discussed further below, the processor 62 may be alsoconfigured to asynchronously send text pattern input data associatedwith a text pattern event to the client computing device 64 forrendering prior to rendering of corresponding contents of the framebuffer 65, to even further enhance user experience. That is, the clientcomputing device 64 may pre-render or preview the changing text evenbefore the corresponding graphics content from the frame buffer 65 isfully received. More particularly, the text pattern input data mayinclude text data (e.g., ASCII or UTF-8 characters, symbols, etc.) andassociated text metadata, such as font type, font size, foregroundcolor, background color, italic, bold, underline, line spacing, etc. Instill another example implementation, the text pattern input data maycorrespond to a text pattern within a scroll container (e.g., a textwindow with a scroll bar).

Furthermore, the processor 62 may also be configured to send thecontents of the frame buffer 65 corresponding to the regions of interestwith a different level of FEC than other content within the framebuffer, as will be discussed further below. In the illustrated example,the processor 62 is configured to send graphics data corresponding tothe regions of interactive graphics from the frame buffer 65 to theclient computing device 64 on a different channel than the othergraphics content from the frame buffer. Here, the interactive graphicsdata is sent via a reliable channel 68, and the remaining graphics datafrom the frame buffer to the client computing device via a lossy channel69. Moreover, different amounts or types of FEC may be applied to eachof these channels, and the amount of FEC applied may also be varied orotherwise throttled in some instances, as will be discussed furtherbelow.

The foregoing will be further understood with reference to variousimplementation examples which may be provided within the framework shownin FIG. 5, for example. However, it will be understood that thetechniques described herein may be implemented in other systemarchitectures as well. System hints for region detection may include oneor more approaches. A first approach is virtual app window tracking. Forexample, this may be provided by the Citrix CVAD Seamless App SupportVirtual Channel (VC) subsystem, which enumerates and tracks updates totop-level windows. Another approach is virtual app process monitoring,e.g., as provided by process monitoring driver used by the Seamless AppSupport VC subsystem. Still another approach may include detectingpatterns, properties and events retrieved from hosted applications ordesktop environment, e.g., based on Microsoft UI Automation API(successor of Microsoft Active Accessibility API). In particular, theclient API enables applications to interact with controls in otherapplications and retrieve information about them. An agent in the HDXsession may use the client API to retrieve information about hostedapplications and the desktop. The agent may register for specific eventnotifications, and can request that specific UI Automation propertiesand control pattern information be passed into its event handlers. Afurther approach may include network loss percentage, loss standarddeviation, latency, and available bandwidth, which may be measured bythe HDX transport driver, for example. It should be noted that losscomputations may exclude the impact of FEC itself.

User hints may include active user input from a keyboard, mouse, touchor other input devices. Another example is user input at specific UIelements of a hosted application or desktop. For example, this mayinclude a click or tap in a UI element supporting ScrollPattern, orkeyboard input in a UI element supporting TextPattern, etc.

The above noted hints are used as inputs and may be utilized to optimizeinteractive graphics delivery by Thinwire, for example, as describedfurther below. As noted above, Thinwire is the Citrix XenApp VirtualChannel (VC) for graphics remoting between host and client. However, itwill be appreciated that other graphics remoting tools and systems mayalso be used in different embodiments.

With respect to optimization techniques for region detection,classification and update prioritization, providing location hints tothe host Thinwire bitmap match-points algorithm may result in faster andmore efficient (less CPU, memory) cached bitmap detection. Furthermore,providing location hints to the host Thinwire text detection algorithmmay advantageously result in faster and more efficient (less CPU,memory) cached glyphs detection.

A system hint for region detection may be provided by window movement,e.g., from a UI automation client agent running in the virtual sessionand listening to WindowPattern events and property updates. This may beused as a trigger to the Thinwire graphics consumer module in thevirtual session. Example parameters may include old and new windowposition, size.

The following is an example process flow which may be used for windowmovement detection and graphics remoting optimization:

-   -   Local screen-to-screen copy/BitBlit (approximate) command sent        server to client.    -   Local fill of exposed area at client:        -   With solid color;        -   Or with pre-cached graphics of exposed window or desktop            background. This may require tracking of window hierarchy            and proactively sending window-surface graphics of            overlapped windows to be cached at client.    -   <asynchronously>    -   Send exposed area server to client.    -   <asynchronously>    -   Provide location hint (region movement) to host Thinwire bitmap        match-points algorithm.        -   Send new window area as now seen by Thinwire:            -   Cached bitmap or new bitmap depending on match hit or                miss respectively.                Note that this approach may also cover the case where                window contents might have changed while dragging.

With respect to window/container scrolling, this may be similar to thewindow movement approach described above, but for localized scrolledarea as opposed to whole window movement. A system hint of scrolling,for example, may be provided from a UI automation client agent runningin the virtual session and listening to ScrollPattern and/orScrollItemPattern events and property updates. This may be used as atrigger to the Thinwire graphics consumer module in the virtual session.Parameters may include old and new values for whether the UI control ishorizontally and/or vertically scrollable, horizontal and vertical viewsizes, horizontal and vertical scroll percentages. The following is anexample process flow:

-   -   Local screen-to-screen copy/BitBlit (approximate) command sent        server to client.    -   Local fill of exposed area at client with solid color.    -   <asynchronously>    -   Send exposed area server to client.    -   <asynchronously>    -   Provide location hint (region movement) to host Thinwire bitmap        match-points algorithm.    -   Send remaining visible region (viewport) of scroll container as        now seen by Thinwire:        -   Cached bitmap or new bitmap depending on match hit or miss            respectively.            Here again, this may cover the case where scrolled item            contents might have changed while scrolling.

With respect to window/container scrolling with text, an optimizedversion of the window/container scrolling described above may be used.More particularly, the following additional system hints may be used:detecting TextPattern within the exposed area of the visible region(viewport) of the scroll container; and detecting TextPattern within theremaining visible region (viewport) of the scroll container. Thefollowing is an example process flow:

-   -   Local screen-to-screen copy/BitBlit (approximate) command sent        server to client.    -   If TextPattern was detected within exposed area:        -   Send actual text with attributes server to client: font            type, font size, foreground/background color, italic, bold,            underlined, line spacing, etc.        -   Local rendering of received text at client (approximate)        -   <asynchronously>        -   Provide location hint (region) to host Thinwire text            detection algorithm.    -   Otherwise:        -   Local fill of exposed area at client with solid color.        -   <asynchronously>        -   Provide location hint to host Thinwire bitmap match-points            algorithm.    -   Send exposed area server to client.        -   If TextPattern actually existed within exposed area:            -   Set of cached and/or new glyphs depending on match hits                or misses respectively        -   Otherwise:            -   Cached bitmap or new bitmap depending on match hit or                miss respectively.    -   <asynchronously>    -   If TextPattern was detected within remaining visible region:        -   Provide location hint (region) to host Thinwire text            detection algorithm.    -   Otherwise:        -   Provide location hint (region movement) to host Thinwire            bitmap match-points algorithm.    -   Send remaining visible region as now seen by Thinwire:        -   If TextPattern actually existed within remaining visible            region:            -   Set of cached and/or new glyphs depending on match hits                or misses respectively.        -   Otherwise:            -   Cached bitmap or new bitmap depending on match hit or                miss respectively.            -   May cover the case where scrolled item contents might                have changed while scrolling, although less likely.

The client first provides an approximate update to the user based on theactual text with attributes (text metadata) received from the server(asynchronously, in practice sooner). The client then provides a truefinal application update using text glyphs (graphics) (asynchronously,in practice layer). This approach advantageously improves interactivityby first providing the light-weight approximate feedback, followed bythe heavier feedback in terms of computing power and bandwidth, yet withaccurate final application feedback. The text metadata may be sent overthe lossy transport 69 and/or with FEC to achieve a furtherinteractivity benefit.

With respect to text editing, a system hint of input focus in UI elementcontaining TextPattern (e.g., from the UI automation client agent 70running in the virtual session and monitoring for input focus intocontrols containing text) may be used as a trigger to the Thinwiregraphics consumer module in the virtual session. Parameters may includelocation of input focus, text within certain range of characters, words,lines, paragraphs, pages or whole document, font type, font size,foreground/background color, italic, bold, underlined, line spacing,etc. The following is an example process flow:

-   -   Determine cursor (input focus) position.    -   Detect input focus in UI element containing TextPattern.    -   Detect either one or more of:        -   Active user input (from input channels)        -   Changing text (TextChangedEvent)        -   Text selection change (TextSelectionChangedEvent) Etc.    -   Provide location hint (region) to host Thinwire text detection        algorithm.    -   Determine important area around input focus:        -   Based on threshold of surrounding word or character count            (before/after), font type and size, line spacing,            indentation, bullet style, text selection range, etc.        -   Request QoS in VC Write interface:            -   Higher priority.            -   Send with high degree of FEC.        -   Send either reliably with FEC or in a lossy manner with FEC.            A decision may be based on where the updated text is            relative to the cursor position:            -   If left/above: not expected to change soon, so use                reliable transmission.            -   If right/below: expected to change soon, so use lossy                transmission until changes stop, then use reliable                transmission.                If the size of text area changed is above a certain                threshold, the technique described above for                window/container scrolling with text may be used to                speed up response time as follows:    -   Send actual text with attributes server to client: font type,        font size, foreground/background color, italic, bold,        underlined, line spacing, etc.    -   Local rendering of received text at client (approximate).    -   <asynchronously>    -   Send important area around input focus.

As with the window/container scrolling with text described above, theclient computing device 64 may first provide approximate updates to theuser through the GUI 66 based on the actual text with attributes (textmetadata) received from the virtualization server 61 (asynchronously, inpractice sooner). The client computing device 64 may then provide a truefinal application update using text glyphs (graphics) (asynchronously,in practice later). This approach advantageously improves interactivityby first providing the light-weight approximate feedback, followed bythe heavier (in terms of computing power and bandwidth) but accuratefinal application feedback. Here again, the text metadata may be sentover the lossy transport 69 and/or with FEC to get further interactivitybenefit.

Referring now to the flow diagram 80 of FIG. 8, related method aspectswith respect to the system 60 are now described. Beginning at Block 81,the method may include hosting virtual computing sessions 63 at thevirtualization server 61 to be remotely displayed at the clientcomputing device 64 via the frame buffer 65, at Block 82. As notedabove, the client computing device 64 renders the virtual computingsessions 63 locally via the GUI 66. The method further illustrativelyincludes detecting regions of interactive graphics within a virtualcomputing session 63 at the virtualization server 61 based upon avirtual session input (e.g., the user and/or system hints describedfurther above) and without analyzing the frame buffer 65, at Block 83,and assigning a higher priority to the regions of interactive graphicswithin the frame buffer at the virtualization server (Block 84). Themethod also illustratively includes sending the contents of the framebuffer 65 from the virtualization server 61 to the client computingdevice 64 for rendering in the GUI 66 based upon the assigned priority(Block 85), as discussed further above. The method of FIG. 8illustratively concludes at Block 86.

Turning to FIGS. 9-10, another example embodiment is now described whichrelates to choosing an appropriate degree and/or type of FEC based uponQoS levels or parameters, and in some instances across different virtualgraphics channels (e.g., reliable and lossy channels). In theillustrated example, a computing system 90 illustratively includes avirtualization server 91 with a processor 92 configured to host virtualcomputing sessions 93 to be remotely displayed via a frame buffer 95,and a client computing device 94 configured to remotely access thevirtual computing sessions from the virtualization server and render thevirtual computing session via a GUI 96, as discussed above. However, theprocessor 92 may be further configured to generate FEC data forgraphical content within the frame buffer 95 based upon a ratio ofgraphical content bandwidth to FEC bandwidth.

Moreover, the processor 92 further sends the graphical content andassociated FEC data to the client computing device 94 over one or morevirtual channels 97 for display within the GUI 96. Additionally, theprocessor 92 is also configured to advantageously determine a QoSparameter associated with the virtual channel(s) 97, and selectivelychange the ratio of graphical content bandwidth to FEC bandwidth basedupon changes in the QoS parameter, as will be discussed further below.

More particularly, the graphical data content sent to the clientcomputing device 94 may be divided into packets, and in one exampleimplementation the QoS parameter may be a packet loss rate. In thiscase, the ratio of graphical content bandwidth to FEC bandwidth maydecrease as the packet loss rate increases, and vice-versa. That is,more FEC data may be generated and sent to the client computing device94 (consuming more of the overall available bandwidth) to help alleviateincreased packet loss. It should be noted that in some cases theprocessor 92 may selectively change the ratio of graphical contentbandwidth to FEC bandwidth without changing an overall bandwidth of thevirtual channel(s) 97, although the processor 92 may also increase/ordecrease the overall available bandwidth while also adjusting the ratioof graphics to FEC data that is being sent. Further details and examplesof how the ratio may be changed are discussed further below

In the example embodiment illustrated in FIG. 10, both a reliablechannel 98 and a lossy channel 89 are used for sending differentgraphics content, such as with the approach described above. In thiscase, different ratios may be used for the graphics data being sentacross the different channels 98, 99. For example, the ratio ofgraphical content bandwidth to FEC bandwidth on the reliable channel 98may higher than a ratio of graphical content bandwidth to FEC bandwidthon the lossy channel 99, as there should be less packet loss across thereliable channel. Further, these ratios may be adjusted or changed basedupon respective QoS measurements associated with each channel, and insome cases different types of QoS measurements may be used for thedifferent channels as well.

The manner in which the QoS measurements are made may different with thedifferent types of measurements. For example, as noted above anddiscussed in further detail below, loss, latency, estimated bandwidth,etc. may be measured by the HDX stack in the above-described Citrixarchitecture. Other measurements, such as expected bit-rate, may bespecific to remoted technology. For example, they may be a factor ofuser input and respective expected graphics subsystem generatinggraphics. A goal may be to use FEC sparingly but sufficiently to ensure“important” data gets delivered without delay and even “less important”data gets delivered with acceptable loss in quality.

In another example implementation, the QoS parameter may relate tolatency, and the ratio of graphical content bandwidth to FEC bandwidthmay decrease as latency increases. That is, more FEC data is sentrelative to the graphics data as latency increases, and vice-versa. Instill another example, the QoS parameter may relate to an estimatedbandwidth for the graphics channels 97, and the ratio of graphicalcontent bandwidth to FEC bandwidth may increase or decrease as theestimated bandwidth increases to allow for more graphics data or FECdata to be sent, as desired. Additionally, in another exampleimplementation, the QoS parameter may comprise an expected bitrateassociated with the virtual channels 97, and the ratio of graphicalcontent bandwidth to FEC bandwidth may decrease as the expected bitratedecreases, for example, and vice-versa. Still another QoS parameter mayrelate to a server load (e.g., how many processes are waiting in thequeue to access the computer processor) associated with thevirtualization server 91.

It should be noted that other QoS parameters than the examples providedabove may be used-in different embodiments, and also that the processor92 may take into account more than one QoS parameter in determining whento make changes to the ratio of graphics content to FEC bandwidth.Furthermore, in some implementations a delay or hysteresis may be usedwhen changing the ratio to account for brief changes or spikes in aparticular QoS measurement, and the changes may be made as the measuredQoS reaches different thresholds, if desired.

The foregoing will be further understood with reference to variousimplementation examples. Here again, these example implementations areprovided in the context of the framework illustrated in FIG. 5, butthese techniques may be applied within other virtualizationarchitectures as well. Generally speaking, it may be desirable to applyFEC when there is packet loss. Otherwise bandwidth may be wasted with nobenefit. The packet loss calculation may be independent, i.e., notskewed by, FEC. By way of example, FEC may be desirable in networkconditions where: there is inherent random packet loss, e.g., on WAN oron wireless network; available bandwidth is high, although it couldstill be limited and shared; and maximum acceptable latency is limited,and also comparable to or lower than the network RTT.

The virtualization server 91 may decide how to split total throughputbetween content bandwidth and FEC bandwidth. For example, the server maydecide to use a 90% to 10% ratio with moderate loss rate, or 66% to 33%ratio with higher loss rate. Other gradients could also be used.

The virtualization server 91 may also simply add FEC to the outgoingstream, which will increase overall bandwidth usage. Alternatively, theserver may keep overall bandwidth usage the same and adjust or otherwisethrottle down the content data rate, while using the remainder for FEC.The latter approach is more conducive to network fair-sharing andcongestion avoidance where the server estimates available bandwidth andensures it does not exceed it.

The virtualization server 91 may package content to be transmitted intogroups (or frames) of ideally same-size packets, where the group size inbytes could be the content bandwidth times maximum acceptable latencydue to FEC computations. For example, for each group of N contentpackets, the FEC algorithm may create K FEC packets. Any reasonablevalues for N and K may be used. The resulting bandwidth split is N/(N+K)for content, K/(N+K) for FEC. Moreover, up to K packets may be lost in agroup transmission, either content or FEC, and the receiving side willstill be able to restore the original N content packets.

At the server transport driver 59, the ICA packet provides a goodnatural boundary, e.g., ˜5 KB in high throughput mode. It may be splitinto 4 packets, for example. In such case, FEC may be applied such that:1 extra FEC packet—tolerate 1 dropped packet (20% loss); and 2 extra FECpackets—tolerate 2 dropped packets (33% loss). In practice, results maybe less as loss is sometimes consecutive.

A simple form of FEC is “exclusive OR” (XOR), which does not requireadditional libraries. There are multiple different forms of FEC andlibraries which may be used in example embodiments, which normally havetwo parts: mathematical computations, and applying the FEC. For example,Jerasure, Erasure Coding Library, is an open source library for thenecessary math primitives. Another example approach is GF-Complete.Other suitable approaches may also be used.

A server graphics consumer module that generates graphics updates to betransmitted may attempt to generate data packets consistent with themaximum application payload size, i.e., the network Maximum TransferUnit (MTU) minus any protocol overhead, e.g., from transport encryption,reliability, framing, ICA protocol overhead, etc. The graphics modulemay generate updates periodically. For example, a frame update will verynaturally map to a group of N content packets protected by a group of KFEC packets. For desired results, to minimize latency, the frame sizemay be equal to, or a multiple of, the maximum application payload size.Similarly, for video encoding, e.g. H.264, the Network Abstraction LayerUnits (NALUs) may be consistent with the maximum application payloadsize.

Latency in the FEC encoding/decoding process in the transport drivers 59would come from having to buffer a complete group of packets at thesender/server before generating FEC, and from restoring lost packets atthe receiver/client side. With the optimization above, latency wouldonly be incurred by having to collect the whole frame update at theclient before starting to render it.

Example embodiments may utilize one or more FEC optimization techniques.A first optimization technique involves graphics transmission over areliable transport or channel 98 with FEC, and/or over a lossy transportof 99 with FEC. In this case, the graphics module may detect EDT astransport, which may provide EDT-reliable and EDT-lossy capabilities.The graphics module may further detect exceeding QoS thresholds of lossrate, latency, available bandwidth, etc. (assuming lowering effectivecontent bitrate is not desired), and it may further switch to a hybridreliable-lossy mode.

Regarding lossy transport with FEC, some graphics updates are sentunreliably, e.g., using an additional Lossy VC Write interface that doesnot guarantee reliable or in-order delivery. Further, FEC requests maybe handled by the transport driver 59 based upon the QoS parameter tothe Lossy VC Write interface. The Lossy VC Write interface returns aticket that can be queried about the state of the packet beingtransmitted, e.g., delivered, lost, unknown, etc. The graphics modulemay optionally hold on to the transmitted data in case it may decide toretransmit it.

In the event that the FEC fails, a response is sent over EDT transportresponse channel, e.g., a Nack/Ack for the packet. The state of theticket associated with the packet is updated, and the graphics moduleholding the ticket receives the update. Furthermore, the graphics moduleretransmits the lost data with the same QoS request, requests moreaggressive FEC, sends it over the reliable transport instead, or decidesto give up on the lost data and sends more recent data instead.

With respect to reliable transport with FEC, other graphics updates arestill sent reliably, but this time using a Reliable VC Write interfacewith a QoS parameter instructing the transport driver 59 to use FEC. Inthe event the FEC fails, EDT transport handles the retransmissionautomatically just like any other packet sent reliably without FEC. Thegraphics module is not concerned with querying the state of the packet,i.e., no ticket needs to be provided.

It should be noted that loss rate and latency thresholds triggering theuse of FEC may be different for reliable vs. lossy transmissions. Forexample, thresholds could be higher for reliable transmissions.Furthermore, the QoS parameter may also imply different FEC algorithmsto use for different configurations, e.g., assuming tradeoffs with CPUand memory. Moreover, the degree of requested FEC for both reliable andlossy transmissions depends on a number of inputs, as will be discussedin further detail below.

Another optimization technique relates to the inputs for determining thedegree or aggressiveness of FEC to be applied. With respect to loss, ahigher FEC may be used for higher average percentage loss and higherstandard deviation of loss. With respect to latency, a higher FEC may beused for higher latency in combination with loss. With respect tobandwidth, a higher FEC may be used for higher estimated bandwidth.

Furthermore, the importance of the graphics data may also be taken intoaccount. More particularly, higher FEC may be used for more importantdata. For example, more important data might relate to key frames ofvideo, adding large bitmap to client cache, drawing bitmap from clientcache, adding glyphs (for text) in client cache, drawing glyphs fromclient cache, solid fills, screen-to-screen copy, text being activelyedited in proximity, etc. On the other hand, less important graphicsdata may relate to intermediary frames of video, transient bitmaps, textthat is being actively scrolled, etc. Similarly, the size of thegraphics data may also be taken into account, e.g., higher FEC forsmaller data. With respect to small packets, it may be most efficient tosimply duplicate them. As noted above, other examples include bitrate(e.g., higher FEC may be used with smaller expected bitrate), serverload (e.g., a higher FEC may be used with more spare capacity), etc.

Still another FEC optimization approach relates to the frequency, type,and/or location of user input. The following are examples: typing fast,font is large; mouse or touch dragging an area of a scrollbar, or awindow title bar; clicking or tapping window control area (minimize,maximize, close); alt-tab or other combination to switch apps, etc.These may indicate a higher immediate bitrate from the graphicsresponse.

Another factor is the application in focus. For example, a web browseror a media player may indicate a higher bitrate than a word processingapp or a simple text editor. Other factors may include top-level windowsize, display resolution, and the graphics subsystem consumer moduleconfiguration (e.g., Frames per Second (FPS)).

In some embodiments, an application of machine learning (ML) may beimplemented using the above-mentioned input to train an ML model. Thismay be done in real time by offloading to a separate appliance. Once anML model has been run on multiple workloads, then simplified staticranges may be created and used for more efficient and real-timeprediction of bitrate.

With regard to bitrate, if the expected bitrate increases, the degree ofFEC may be decreased, and optionally turned off above a threshold level.This increases the effective content bitrate, but with an associatedrisk of effective loss. However, this may be desirable becausesubsequent updates may supersede any lost graphics updates. In addition,bandwidth savings may be desired to avoid reaching estimated bandwidthand causing congestion.

On the other hand, if expected bitrate decreases, then the degree of FECmay be increased. If insufficient packets are available in the sendqueue to complete a group/batch of FEC-encoded packets, the transportdriver 59 may send dummy packets to avoid delay, or duplicate packets.This decreases effective content bitrate but also decreases risk ofloss. Moreover, this is desired where subsequent updates are notexpected soon. In addition, sufficient bandwidth may be expected.

Below an expected bitrate threshold level, the graphics consumer modulemay send a high-quality update. For example, if user input stops and nomore changes are expected, a “final/sharp” image may be sent. Current“build to lossless” systems send the high-quality update when detectedchanges stop, which invariably happens with some delay and detracts frominteractivity. Using hints such as user input allows the system 90 todetect the need to send a high-quality update sooner and improvesinteractivity. The high-quality update may be sent reliably with FEC,for example.

Referring additionally to the flow diagram 210 of FIG. 11, relatedmethod aspects with respect to the system 90 are now described.Beginning at Block 211, the method illustratively includes hostingvirtual computing sessions 93 at the virtualization server 91 to beremotely displayed at the client computing device 94 via the framebuffer 95, at Block 212. The method further illustratively includesgenerating FEC data for graphical content within the frame buffer 95 atthe virtualization server 91 based upon a ratio of graphical contentbandwidth to FEC bandwidth, and sending the graphical content andassociated FEC data from the virtualization server to the clientcomputing device 94 over virtual channel(s) 97 for display within theGUI 96, at Block 213. The method also illustratively includesdetermining a QoS parameter associated with the virtual channel(s) 97,at Block 214, and selectively changing the ratio of graphical contentbandwidth to FEC bandwidth at the virtualization server 91 based uponchanges in the QoS parameter, at Blocks 215, 216, as discussed furtherabove.

Turning to FIGS. 12-13, another example embodiment is now describedwhich relates to prioritize and divide user input data and send itacross different virtual channels to provide enhanced graphics renderingcapabilities. A computing system 220 illustratively includes avirtualization server 221 configured to host virtual computing sessions223, and a client computing device 224 configured to render the virtualcomputing sessions within a GUI 226, as discussed above. The clientcomputing device 224 illustratively includes a processor 230 configuredto receive a stream of user input data associated with the virtualcomputing session from one or more user input devices 231, and classifythe stream into first and second data packets.

More particularly, the first data packets have a higher priority thanthe second data packets. The processor 230 is also configured to sendthe first (i.e., high priority or critical) data packets to thevirtualization server 221 via a first (e.g., reliable) virtual channel228, and send the second (lower) data packets to the virtualizationserver via a second virtual channel 229 having a higher packet loss rateassociated therewith than the first virtual channel (e.g., a lossychannel). The virtualization server 221 illustratively includes aprocessor 222 that is configured to assemble the second data packets toreconstruct and inject the stream of user input data into the virtualcomputing session 223 based upon the first data packets.

In the example illustrated in FIG. 13, a string of data packets 1through N are included in the stream of user input, and the first datapackets (here packets 1 and N) are the beginning and ending packets ofthe stream of user input data, respectively. The second data packets arethose in between the first and last packets, namely the packets 2through N-1. In this way, the first data packets advantageously definepoints of reference or anchor points within the stream of user inputdata, and the client computing device 224 may send the second datapackets to the virtualization server 221 via the second virtual channel229 along with references to the anchor points (e.g., that the startpacket is packet 1 and the end packet is packet N). As such, theprocessor 222 may advantageously assemble the second data packets (i.e.,packets 2 through N-1) to reconstruct and inject the stream of userinput data into the virtual computing session 223 based upon thereferences to the anchor points (i.e., start and end data packets 1 andN, respectively). It should be noted that other approaches for selectinghigh priority packets and referencing these packets may be used indifferent embodiments.

In some example embodiments, the processor 230 may be further configuredto also send one or more of the first data packets via the secondvirtual channel 229 as well as the first data channel 228 to provideredundancy and help ensure faster reconstruction of the user input datastream should communications over the first channel 228 be delayed, ifdesired. Also, as discussed further above, the processor 230 may alsoadvantageously apply FEC to the first and second data packets prior tosending, and also selectively change a level of FEC applied to one orboth of the first and second data packets based upon spikes in datapacket loss, etc., if desired.

The user input events creating the stream of upper input may come fromvarious different types of user input devices 231. For example, wherethe user input device 231 is a keyboard, the first data packets maycorrespond to keyboard events input via the keyboard. Where the userinput device 231 is a touch screen, the first data packets maycorrespond to touch-up and touch-down events (e.g., when a fingertouches or is removed from the display) input via the touch screen,while the second data packets may correspond to touch-move events (e.g.,when a finger touching the display is moved along the display) input viathe touch screen, for example. In another example embodiment where theuser input device 231 is a mouse, the first data packets may correspondto mouse-button-up and mouse-button down events (e.g., which the mousebutton is depressed or released) input via the mouse, while the seconddata packets may correspond to mouse-drag (e.g., the mouse button isheld and/or the pointer moved) or mouse-wheel events (e.g., the mousewheel is scrolled up or down) input via the mouse. In still anotherexample embodiment where the user input device 231 is an electronic penor stylus, the first data packets may correspond to pen-up and pen downevents (e.g., when the electronic pen touches or is removed from thedisplay) input via the electronic pen, while the second data packets maycorrespond to pend-drag events (e.g., when the electronic pen istouching the display and the point is moved along the display) input viathe electronic pen. However, it should be noted that other input devicesand designations of the first (high priority) and second (lowerpriority) data packets may be used in different embodiments.

The foregoing will be further understood with reference to variousimplementation examples. Here again, these example implementations areprovided in the context of the framework illustrated in FIG. 5, butthese techniques may be applied within other virtualizationarchitectures as well. Above certain thresholds of loss and latency, oneapproach which may be used is to send important input events, e.g.keyboard events, touch-down, touch-up, mouse-button-down,mouse-button-up, etc. reliably. Moreover, less important input eventsare sent in a lossy way, e.g. touch-move, mouse-drag, mouse-wheel,pen-drag. In addition, FEC or simple packet duplication may optionallybe applied to one or both reliable and lossy user input transmissions,since impact to the bandwidth will be negligible but the redundancy willguarantee smooth user experience in conditions of loss and latency. Thisis consistent with low expected bitrate, small data size and importantdata factors discussed above.

As noted above, user input may be prioritized and classified intocritical and non-critical input (packets) and sent over reliable andnon-reliable (lossy) streams 228, 229 respectively. Since the reliableand lossy streams are independent and asynchronous, reliable versuslossy input packets may be received out of order at the host, which maybreak the semantics of user input. Therefore, for each non-criticallossy packet the client computing device 224 may identify a prerequisitecritical reliable packet by sequence number. This is specified at theclient computing device 224 at time of packet formation. A criticalpacket may identify an “opening” of input sequence, e.g., mouse downbefore dragging begins.

Lossy packets may be cached at the host and not replayed (injected intothe hosted app/desktop) until the respective critical prerequisitepacket is also received. When a sequence “closing” critical packet isreceived, e.g., a mouse-button-up, any non-critical lossy packet thatmay be received subsequently and referring to a now “closed”prerequisite is discarded. For example, this may occur with a mouse dragthat may be discarded if it is received after the sequence-closingmouse-button-up is received.

Critical input opportunistically may be sent over both reliable andlossy streams in some implementations. This further reduces potentialsynchronization lag on the host side when an input sequence involvingreliable and lossy packets starts. For example, a mouse down event maybe sent over both reliable and lossy transports. As explained above,mouse drags are held at the host before the correspondingsequence-starting mouse down is received first. Advantageously, if themouse down is delivered as a prerequisite along with the mouse drags,the host can immediately start playing/injecting the input sequence.This may be useful if, for example, reliable and lossy streams arerouted differently, different QoS is applied on them while in transit,and/or packets tend to arrive out of order for other reasons.

Additional FEC may be performed with some delay as well to handle spikesin loss. The delay with which this additional FEC (or packetduplication) is applied may be computed based on at least one of anaverage or standard deviation of loss spikes duration. In modernnetworks spikes of loss may occur, e.g., due to stochastic interference,where loss is consecutive rather than spread out and consistent with anaverage loss percentage. For example, in a network with 5% averagepacket loss, there may be spikes of loss where loss is at 50% for aduration of 2 seconds. So even though the initial FEC may be substantialand sufficient to cover 5% packet loss, it may be insufficient to coverthe 20% loss during the spike. Therefore, applying an additional FEC (orduplication) after 2 seconds will improve the chances of the packetbeing delivered, although in some cases with delay. This may be betterthan applying the additional FEC consecutively, because a consecutiveapplication may coincide with a spike.

Adjusting or throttling of the FEC may be performed using approachessimilar to those discussed above. For example, the client computingdevice 224 may monitor typing of words or sentences and detectincomplete phrases or grammar to determine (anticipate) with a highlevel of confidence additional immediate user input. In other words, theclient computing device 224 may predict incomplete words or sentence,and tune the extent of FEC accordingly (e.g., greater FEC for moredropped words/sentences).

Where size of the input is typically small, FEC has only a negligibleimpact on bandwidth and CPU but, in combination with reliable and lossytransmission, has significant impact on interactivity. Test results anddemo videos were performed at 300 ms RTT with 5% packet loss.Approximately 3× improvement in interactivity was achieved with windowmovement, scrolling, painting, and other types of interactions, relativeto the same interactions over an EDT reliable transport alone.

FEC may be applied to interactive traffic only in some embodiments,e.g., real-time audio or multimedia VCs, similar to Thinwire graphicsdiscussed above. Bulk data transfer VCs, e.g., Client Drive Mapping (CDMVC, typically need not use the benefits of QoS requesting FEC, thusminimizing impact to bandwidth, although FEC may similarly be used insome embodiments if desired, for example, to increase the speed ofremote file transfer at the expense of additional bandwidth usage.

Turning to the flow diagram 140 of FIG. 14, method aspects related tothe system 220 are now described. Beginning at Block 141, the methodillustratively includes remotely accessing a virtual computing session223 at the client computing device 224 from the virtualization server221, at Block 142, and receiving a stream of user input data at theclient computing device 224 associated with the virtual computingsession and classifying the stream into first and second data packets,at Block 143. As noted above, the first data packets have a higherpriority than the second data packets. The method also illustrativelyincludes sending the first data packets from the client computing device224 to the virtualization server 221 via the first virtual channel 228,at Block 144, and sending the second data packets from the clientcomputing device to the virtualization server via the second virtualchannel 229 which has a higher packet loss rate associated therewiththan the first virtual channel, at Block 145. As a result, thevirtualization server 221 may advantageously assemble the second datapackets to reconstruct and inject the stream of user input data into thevirtual computing session 223 based upon the first data packets, asdiscussed further above. The method of FIG. 14 illustratively concludesat Block 146.

Many modifications and other embodiments will come to the mind of oneskilled in the art having the benefit of the teachings presented in theforegoing descriptions and the associated drawings. Therefore, it isunderstood that the foregoing is not to be limited to the exampleembodiments, and that modifications and other embodiments are intendedto be included within the scope of the appended claims.

That which is claimed is:
 1. A computing device comprising: a memory anda processor cooperating with the memory to host virtual computingsessions to be remotely displayed at a client device via a frame buffer,the client device configured to render the virtual computing sessionsvia a graphical user interface (GUI), classify graphics data, based upona virtual session input, to be stored within the frame buffer asinteractive graphics data defining regions of interactive graphics orother graphics data; assign a first priority to the interactive graphicsdata and a second priority to the other graphics data, the firstpriority being higher than the second priority, and send the graphicsdata from the frame buffer to the client computing device for renderingin the GUI so that the interactive graphics data defining regions ofinteractive graphics and having the first priority is rendered beforethe other graphics data having the second priority.
 2. The computingdevice of claim 1 further configured to capture the virtual sessioninput.
 3. The computing device of claim 1 wherein the virtual sessioninput comprises a window pattern event.
 4. The computing device of claim1 wherein the virtual session input comprises a scroll pattern event. 5.The computing device of claim 1 wherein the virtual session inputcomprises a text pattern input data.
 6. The computing device of claim 5wherein the processor is further configured to asynchronously send thetext pattern input data to the client device for rendering prior torendering of corresponding graphics data from the frame buffer.
 7. Thecomputing device of claim 5 wherein the text pattern input datacomprises text data and associated text metadata; and wherein the textmetadata comprises at least one of font type, font size, foregroundcolor, background color, italic, bold, underline, and line spacing. 8.The computing device of claim 5 wherein the text pattern input datacorresponds to a text pattern within a scroll container.
 9. Thecomputing device of claim 1 wherein the processor is configured to sendthe interactive graphics data within the frame buffer having the firstpriority with a different level of forward error correction (FEC) thanthe other graphics data within the frame buffer having the secondpriority.
 10. The computing device of claim 1 wherein the processor isconfigured to send the interactive graphics data within the frame bufferhaving the first priority via a different virtual channel than the othergraphics data within the frame buffer having the second priority. 11.The computing device of claim 10 wherein the processor is configured tosend the interactive graphics data having the first priority from theframe buffer to the client computing device via a first channel, andsend the other graphics data having the second priority from the framebuffer to the client computing device via a second channel.
 12. Thecomputing device of claim 1 wherein the processor classifies thegraphics data without analyzing data within the frame buffer.
 13. Amethod comprising: hosting virtual computing sessions at a computingdevice to be remotely displayed at a client device via a frame buffer,the client device being configured to render the virtual computingsessions via a graphical user interface (GUI); classify graphics data,based upon a virtual session input, to be stored within the frame bufferas interactive graphics data defining regions of interactive graphics orother graphics data; assigning a first priority to the interactivegraphics data and a second priority to the other graphics data at thecomputing device, the first priority being higher than the secondpriority; and sending the graphics data from the frame buffer to theclient device for rendering in the GUI so that the interactive graphicsdata defining regions of interactive graphics and having the firstpriority is rendered before the other graphics data having the secondpriority.
 14. The method of claim 13 wherein the virtual session inputcomprises a window pattern event.
 15. The method of claim 13 wherein thevirtual session input comprises a scroll pattern event.
 16. The methodof claim 13 wherein the virtual session input comprises a text patterninput data.
 17. A computing system comprising: a server configured tohost virtual computing sessions and remotely display the virtualcomputing sessions via a frame buffer; and a client device configured toremotely access virtual computing sessions from the server via agraphical user interface (GUI); wherein the server is further configuredto classify graphics data, based upon a virtual session input, to bestored within the frame buffer as interactive graphics data definingregions of interactive graphics or other graphics data; assign a firstpriority to the regions of interactive graphics data and a secondpriority to the other graphics data, and send the graphics data from theframe buffer to the client device for rendering in the GUI so that theinteractive graphics data defining regions of interactive graphics andhaving the first priority is rendered before the other graphics datahaving the second priority.
 18. The computing system of claim 17 whereinthe server is further configured to capture the virtual session input.19. The computing system of claim 17 wherein the wherein the virtualsession input comprises a window pattern event.
 20. The computing systemof claim 17 wherein the wherein the virtual session input comprises ascroll pattern event.
 21. The computing system of claim 17 wherein thewherein the virtual session input comprises a text pattern input data.